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Introduction 

Today,  knowledge-based  systems  constitute  a  new  and  exciting  way  to  bring  the 
best  of  medical  knowledge  to  students,  practitioners  and  patients.  Expert  systems 
applications  have  been  implemented  in  a  variety  of  settings.  A  few  examples  of  medical 
expert  systems  include  Iliad,  HELP,  DXplain,  CARE  and  QMR,  PUFF,  Oncocyn, 
etc. (refs).  The  performance  of  these  systems  has  been  documented  in  the  literature  and 
judged  at  least  as  good  as  that  of  human  experts.  However,  only  a  few  have  made  it  to  the 
real  world  and  are  commercial  available. 

Today,  the  field  and  society  at  large  need  to  know  how  artificial  intelligence 
applications  and  expert  systems  in  particular  will  make  a  difference  in  the  real  world.  Thus, 
a  major  issue  for  developers  of  expert  systems  is  to  investigate  practical  applications  of  the 
technology.  One  interesting  such  application  constitutes  the  substance  of  the  collaboration 
between  the  US  ARHD  and  Applied  Informatics.  The  collaboration  aims  at  the  development 
of  a  prototype  diagnostic  expert  system  in  infectious  diseases  which  might  be  rare  in  the 
United-States  but  prevalent  in  foreign  countries.  The  purpose  of  this  work  is  to 
demonstrate  the  feasibility  of  a  prototype  which  is  envisioned  to  operate  in  remote  places 
where  medical  expertise  about  the  rare  diseases  might  be  scarce  and  the  handling  of  cases 
enhanced  with  the  use  of  such  a  diagnostic  expert  system. 

Qfever  was  selected  as  the  infectious  disease  agent  for  studying  the  feasibility  of  a 
prototype  diagnostic  expert  system.  Qfever  is  a  ricketsiella  infectious  disease  which, 
although  not  lethal,  causes  severe  morbidity.  The  chronic  form  of  the  disease  may  result 
from  the  primary  infection  and  lead  to  cardiac  and  liver  involvement.  The  medical  expert 
system  used  in  this  project  is  the  Iliad  program  developed  by  Applied  Informatics  and  the 
University  of  Utah.  Iliad's  current  knowledge  base  contains  over  1090  diseases  and 
syndromes  across  all  of  the  domains  of  internal  medicine.  The  primary  focus  of  Iliad  is  to 
teach  and  test  clinical  problem-solving  skills  and  provide  a  wide  range  and  variety  of 
simulated  patient  cases. 

The  project  started  in  January  1990.  Phase  1  of  the  project  describes  the  diagnostic 
criteria  for  Qfever  using  the  literature,  expert  judgement  and  real  patient  cases.  Phase  2  tests 
the  performance  of  the  prototype  against  real  as  well  as  simulated  cases.  Phases  1  and  2 
repeat  until  a  reliable  performance  evolves.  Another  validation  measurement  and  refinement 
of  the  prototype  was  obtained  through  face-value  evaluation  of  the  medical  logic  by  other 
experts  in  the  field.  Dr.  Thomas  Marrie  from  Nova  Scotia  was  involved  in  critiquing  the 
Qfever  disease  profiles  we  developed  as  well  as  in  the  validation  study  of  the  prototype 
based  on  a  relatively  large  database  of  Qfever  cases  (188  cases). 

After  the  one  year  duration  of  this  work,  the  Iliad  prototype  for  Q  fever  is 
operational  and  the  first  performance  tests  are  encouraging.  The  temporal  dimension  of 
Qfever  was  modelled  in  Iliad  using  two  clinically  significant  states  of  Qfever  (i.e.,  early 
and  late  stages).  These  two  states  were  deemed  by  medical  experts  as  adequate  to  represent 
the  different  facets  of  the  disease.  Furthermore,  a  link  to  a  database  of  digitized  pictures  has 
strengthened  the  educational  value  of  the  prototype  expert  system. 

In  the  future,  there  needs  to  be  further  study  of  the  reliability  fo  the  current  Qfever 
prototype.  This  would  involve  the  abstracting  of  a  large  number  of  cases  and  a  performance 
judgement  of  the  Iliad  prototype  by  independent  experts  Field  testing  is  the  next  ievel  of 
testing.  This  level  of  evaluation  is  the  study  of  the  impact  of  Iliad  on  the  judgement  of  non¬ 
expert  physicians.  Field  testing  is  also  useful  to  collect  data  about  the  transferability  of  the 
program  and  the  program's  performance  in  different  geographical  locations.  Finally,  once 
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this  'vertical'  investigation  of  the  first  prototype  is  achieved  with  satisfactory  results,  the 
knowledge  base  is  expanded  to  include  new  disease  profiles.  The  more  knowledge  the 
system  offers  to  its  users  the  more  useful  and  reliable  the  system  will  be. 


Development  of  a  Prototype  Diagnostic  Expert  System  in 
Infectious  Diseases 

In  general,  the  development  a  prototype  diagnostic  expert  system  starts  with  the 
definition  of  the  knowledge  domain  and  the  intended  audience.  Rare  infectious  diseases  is 
the  knowledge  of  interest  and  Qfever  is  the  specific  agent  selected  to  carry  out  our  study. 
The  task  is  then  to  develop  a  diagnostic  model  of  the  Qfever  infection  combining  all 
available  sources  of  information.  The  model  would  be  tested  in  remote  areas  where  Qfever 
infection  is  prevalent  and  the  medical  staff  is  not  particularly  familiar  with  the  pathology. 


The  knowledge  Engineering  Environment 

The  process  of  knowledge  engineering  in  Iliad  refers  to  the  sum  of  all  activities  that  aim  to 
capture,  restructure  and  validate  knowledge  in  a  particular  domain  in  order  to  build  a  computer 
representation  of  this  knowledge  for  educational,  training  and  testing  purposes.  In  this  section,  we 
will  give  an  overview  of  the  different  steps  involved  in  the  knowledge  engineering  process  which 
is  illustrated  by  the  diagram  below  and  refer  to  specific  chapters  in  the  knowledge  engineering 
handbook  for  in  depth  information  about  each  step  (see  Knowledge  Engineering  in  Iliad:  the 
Development  Tools). 
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Figure  1:  The  knowledge  engineering 


process  in  Iliad 


The  first  step  in  knowledge  engineering  is  the  acquisition  of  knowledge  from 
human  experts.  Knowledge  is  acquired  through  formal  interview  techniques  with  a  medical 
domain  expert  which  is  facilitated  by  a  knowledge  engineering  using  computerized  support 
tools  in  a  quiet,  comfortable  setting. 

Step  2,  in  the  knowledge  engineering  process,  is  to  match  the  free  text  terminology 
used  during  the  frame  development  effort  to  coded  vocabulary  terms  in  the  Knowledge 
Base  Dictionary  (KBD). 

Once  all  the  terms  used  in  a  knowledge  frame  have  been  defined  in  the  KBD  ,  the  Frame 
Author  program  is  used  to  create  Iliad  frames  (step  3).  The  Frame  Author  converts  the  free  text  fo 
knowledge  to  a  structured  form  in  which  every  item  is  an  item  previously  defined  in  the  KBD. 

The  frames  assembled  with  the  Frame  Author  program  are  then  checked  before  they 
can  be  compiled  to  run  with  Iliad.  This  is  done  in  step  4  using  the  IliadUtilities  program. 

Once  the  pre-compilation  checks  have  cleared,  the  BatchCompiler  is  used  to 
compile  the  frames  into  the  "Iliad  files"  folder  (step  5). 

Running  Iliad  to  test  the  performance  of  the  frames  is  the  next  step  (step  6)  in  the 
knowledge  engineering  cycle.  The  knowledge  engineering  process  cycles  with  many  revisions  to 
the  knowledge  base  taking  place  before  a  reliable  performance  of  the  system  evolves. 


The  Qfever  Prototype 
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Qfever  is  the  infectious  disease  agent  selected  to  study  the  feasibility  of  a  diagnostic 
expert  system  that  will  support  military  medical  staff  in  remote  areas.  Qfever  is  a  relatively 
rare  disease  in  the  US,  but  outbreaks  have  been  reported  all  around  the  world.  Serology 
testing  (ELISA  test)  allows  for  the  identification  of  offending  agent:  ricketsiella  Coxiella 
Burnetii.  The  infectious  condition,  when  recognized,  has  been  treated  with  appropriate 
dosages  of  tetracyclin.  However,  the  disease  presentation  is  not  very  specific  and  is  often 
confused  with  other  pathologies  (e.g.,  atypical  pneumonia)  and  left  untreated.  A  fraction  of 
the  acute  Qfever  infections  develop  into  chronic  Qfever  with  liver,  and  heart  involvement. 

Coxiella  Burnetii  is  an  aerosol  and  is  thought  to  constitute  a  potential  bacteriological 
weapon.  However,  in  most  cases,  the  infection  was  traced  to  animal  carriers  who  do  not 
suffer  from  any  consequences.  Specifically,  Coxiella  Burnetii  can  be  contracted  from 
animal's  placenta,  thus  more  prevalent  in  farmers,  slaughter  house  personel,  and  infectious 
diseases  laboratory  personel. 

The  development  of  three  major  frames  for  Qfever:  Qfever,  (Early  Stage),  Qfever 
(Late  Stage)  and  Blood  Culture  Negative  Endocarditis  were  considered  by  experts  in  the 
domain  to  adequately  cover  the  relevant  distinction  between  the  different  phases  of  the 
disease.  Appendix  A  includes  the  disease  profiles  of  these  three  frames  and  the  related 
intermediate  disease-syndromes. 


Temporal  Aspects 

The  temporal  aspect  of  an  expert  system  refers  to  a  set  of  capabilities  that  helps  the 
system  recognize  and  take  advantage  of  time-oriented  clinical  events,  such  as  the  first  value 
of  a  laboratory  test,  the  value  of  a  test  after  or  before  the  occurrence  of  another  event,  and 
the  identification  of  trends.  This  set  of  capabilities  may  help  the  expert  system  deal  with 
disease  stages  and  severities. 

A  natural  way  to  incorporate  temporal  aspects  into  a  decision  support  module  is  to 
tie  the  module  to  a  live  clinical  database.  The  knowledge  base  could  then  be  'time-driven' 
as  well  as  'data-driven'. 

Because  Iliad  is  not  tied  to  real-time  patient  database,  the  only  way  to  a  model  a 
disease  over  time  is  to  build  a  frame  for  each  significant  phase  of  the  diagnostic  process. 
Thus,  in  the  Qfever  example,  domain  experts  have  judged  that  a  good  approximation  of  the 
disease  process  is  accomplished  with  two  frames:  one  for  the  early  or  acute  phase,  and  one 
for  the  late  or  chronic  phase. 


Video  Link 

Any  disease  or  finding  name  in  the  Iliad  knowledge  base  dictionary  can  be  linked  to 
available  digitized  pictures.  At  any  point  during  a  consultation  session  with  Iliad,  the  user 
can  access  the  pictorial  representations  to  leam  more  about  the  diagnostic  criteria  or  patient 
finding. 

Iliad  will  append  a  "(p)"  at  the  end  of  the  text  of  a  term  to  indicate  the  term  has  one 
or  more  pictures  linked  to  it.  Wherever  a  dictionary  term  appears  in  Iliad,  as  a  patient 
finding  or  a  workup  suggestion,  the  user  can  select  the  term  and  activate  the  'Video' 
command  within  the  'Browse'  menu.  Each  picture  comes  with  text  describing  the  content 
of  the  picture.  The  textual  description  of  a  picture  can  include  references  to  other  pictures 
within  the  picture  database,  in  which  case,  the  name  of  the  picture  referenced  is  in  bold. 
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Double  clicking  on  the  bold  name  will  display  the  referenced  picture  in  a  'Hyperpicture’ 
fashion. 

To  access  digitized  pictures  within  Iliad,  the  knowledge  engineer  creates  textual 
descriptions  of  the  pictures  and  links  each  picture  to  one  or  more  Iliad  dictionary  codes  that 
they  illustrate. 

The  hardware  requirements  for  linking  Iliad  to  digitized  pictures  are  the  same  as  for 
running  Iliad  plus  a  color  monitor  (e.g.,  MacII,  MacSI)  for  adequate  picture  quality.  A 
manual  exists  with  instructions  on  how  to  create  and  link  digitized  pictures  to  Iliad  (see 
Linking  Digitized  Pictures  to  Iliad). 


Transferability 

The  term  transferability  in  this  context  refers  to  the  ability  of  the  system's 
knowledge  base  to  perform  reliably  in  different  geographical  locations.  Iliad’s  probabilistic 
knowledge  representation  models  are  particularly  adaptable  for  reflecting  geographical 
specificities.  Indeed,  the  Bayesian  model  used  by  Iliad  includes  the  a  priori  of  a  disease  or 
of  a  diagnostic  finding  in  its  calculations.  These  frequencies  can  be  estimated  from  a  sample 
of  the  local  population.  Furthermore,  Iliad  allows  for  a  site  specific  file  where  disease 
prevalences  are  stored  and  local  knowledge  is  preserved. 

Iliad's  knowledge  base  development  approach  assumes  there  will  only  be  one 
central  site  for  maintaining  the  master  knowledge  base  with  each  user  site  adding  disease 
prevalence  information.  The  central  site  will  integrate  feedback  from  all  users  into  the 
master  knowledge  base  file.  Also,  there  are  tools  for  producing  and 
maintaining  different  versions  of  the  expen  system  in  multiple  languages  (e.g,  French, 
English). 


Testing  the  Qfever  Prototype 

Testing  of  a  diagnostic  expert  system  encompasses  process  and  outcome 
performance  measures.  These  performance  measures  will  validate  the  reliability  of  the 
knowledge  base  as  well  as  the  impact  of  the  expert  system  on  the  outcome  of  the  practice  of 
its  intended  users. 

This  contract  work  did  not  include  any  study  of  outcome  measures.  The 
performance  of  the  knowledge  base  was  measured  in  several  steps.  First,  a  face-value 
evaluation  of  the  disease  descriptions  was  done  by  another  infectious  disease  specialist  (Dr. 
Thomas  Marrie  from  Victoria  General  Hospital  in  Nova  Scotia).  Then,  Dr.  Larry  Reimer 
from  the  University  of  Utah  Medical  Center,  developer  of  the  original  infectious  disease 
profiles,  tested  the  behavior  of  the  system  using  textbook  examples,  case  reports  in  the 
literature  and  the  simulation  mode  of  Iliad.  The  simulation  mode  in  Iliad  allows  for  the 
generation  of  realistic  patient  cases  from  a  disease  profile.  Generating  patient  cases  from  the 
expert  system's  knowledge  base  has  been  compared  to  the  "Turing"  test  for  medical  expert 
system. 

Finally,  we  are  currently  in  the  process  of  testing  the  performance  of  the  system  on 
188  Qfever  cases  from  Nova  Scotia  collected  by  Dr.  Marrie.  Tools  for  transferring  data 
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from  the  large  Nova  Scotia  database  have  been  developed  to  facilitate  the  performance 
study  of  the  prototype  against  real  cases 


Prospects  for  the  Future 

The  validation  of  the  Qfever  prototype  against  a  large  set  of  patient  cases  should 
enhance  the  reliability  of  the  knowledge  base.  Another  extension  of  this  project  would 
involve  field  testing  the  prototype  in  different  geographical  locations.  To  facilitate  the 
transferability  of  the  program,  we  added  to  the  Iliad  program  the  possibility  to  modify  the 
content  of  the  knowledge  base  to  reflect  geographical  prevalence  of  disease.  Finally, 
parallel  efforts  to  the  validation  studies  should  aim  to  expand  the  knowledge  base  to  include 
other  rare  infectious  diseases.  This  would  make  interaction  with  the  expert  system  more 
valuable.  The  personel  of  USARIID  has  been  introduced  to  Iliad  knowledge  engineering 
techniques  and  familiarized  with  the  knowledge  development  tools  in  order  to  facilitate  the 
ongoing  efforts  of  knowledge  development  and  validation. 
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of  the  Qfever  Prototype  Expert  System 

Omar  Bouhaddou,  PhD 

Senior  knowledge  engineer  at  Applied  Informatics 

Homer  R.  Warner,  Jr. 

principal  investigator  and  president  of  Applied  of  Informatics 

Homer  R.  Warner,  MD,  PhD 

professeur,  chaiman  of  the  department  of  medical  informatics  and  knowledge 
engineer  on  the  Iliad  project 
Dean  K  Sorenson,  PhD 

biochemist  and  knowledge  engineer  on  the  Iliad  project 

Stan  Altman,  MD 

hematologist  and  knowledge  engineer  on  the  Iliad  project 

Larry  Reimer,  MD 

infectious  diseases  specialist  and  knowledge  engineer  on  the  Iliad  project 

Thomas  Marrie,  MD 

infectious  diseases  specialist  and  consult  jit  on  the  Qfever  prototype 

Tim  Cannon 

primary  contact  at  the  USARIID 
Glen  Higbee 

principal  investigator  at  the  USARIID 
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Appendix  A:  QFEVER  DISEASE  PROFILES 

A  few  guidelines  on  how  to  read  the  disease  descriptions  below: 

1.  The  disease  name  appears  at  the  top 

2.  the  a  priori  represents  the  prevalence  of  the  disease  in  the  hospital 
(our  hospitals) 

3.  the  findings  are  either  direct  observations  (e.g.,  historical, 
physical  examinations,  laboratory  results)  or  a  group  of  findings 
described  in  a  sperate  frame  and  indicated  by  an  ’ sign. 

4.  sensitivity  is  the  true  positive  rate  =  the  fraction  of  patients 
with  the  disease  that  have  with  the  finding. 

5.  1  ~  specificity  is  the  false  positive  rate  =  the  fraction  of  patients 
without  the  disease  that  have  with  the  finding. 
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Disease  name:  Q  fever  (early  stage) 
A  priori  =  .00003 


findings:  sensitivity  1-specificity 


a. 

Temperature 

>38  and  <  40  C 

.95 

.15 

>=  40  C  (104) 

.52  (1) 

.01 

b. 

@Atypical_Pneumonia 

.51 (5) 

.015 

(7) 

or 

@Non-specif ic  symptoms  of  Q  fever 

.90 

.20 

d. 

Hx  of  headache  at  retrobulbar  area 

.30  (3,10) 

.01 

or 

Ex  of  headache 

.90  (2) 

.15 

or 

Hx  of  headache  [with  a  specific 

.25 

.03 

e. 

quality]  which  is  severe  and  frontal 

Sex,  male 

.85 

.57 

g- 

Coxiella  Burnetii  ELISA  is  positive 

.50  (4) 

.01 

>=  7  days  of  onset  of  symptoms 
or 

Coxiella  Burnetii  ELISA  is  positive  4  fold  rise 

.50  (4) 

.001 

>=  7  days  of  onset  of  symptoms 
or 

Immunofluorescence  shows  Coxiella  Burnetii 

.95  (4) 

.01 

antibody  >=  15  days  of  onset  of  symptoms 
or 

Immunofluorescence  shows  Coxiella  Burnetii 

.90  (4) 

.001 

antiboay  4  fold  rise  >=  15  days  of  onset 
of  symptoms 
or 

Complement  fixation  test  shows  >=  4  fold 

.90  (2,3,4) 

.001 

rise  in  phase  II  antibody  titer  to  C.  Burnetii 
>=  7  days  of  onset  of  symptoms 
or 

Complement  fixation  test  shows  phase  II 
antibody  titer  >  1:128  to  C.  Burnetii 
>=  7  days  of  onset  of  symptoms 

.95  (2) 

.001 

g- 

hx  of  contact  with  cattle,  sheep,  goats 

.70 

.02 

or 

Hx  of  contact  with  sheep  placenta 

.50 

.001 

or 

Hx  of  work  in  mimal  research  facility 
housing  sheep 

.20 

.01 

or 

Hx  of  contact  with  cats 

.50 

.40 

h. 

@Abnormal  liver  functions  criteria  for  Q  fever 

.65 

.05 

or 

PE  shows  hepatomegaly 

.43(1,3,5,6) 

.06 

or 

Liver  tenderness 

.11(6) 

.03 

(9) 

or 

PE  shows  splenomegaly 

.10  (10) 

.01 

a  priori:  about  1  case  /  year 
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?confirm  this  before  adding: 

cxr  of  multiple  rounded  opacities  .95  (10) 
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1.  Spelman  WD.  Q  fever:  a  study  of  111  consecutives  cases  (1982) .  Med  J 
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Abnormal  liver  function  criteria  for  Q  fever 

a .  SGOT  u/mL 

b.  SGPT  u/mL 

c.  alkaline  phosphatase  U/L 

d.  Serum  bilirubin  mg/100mL 

True  if:  a  >  34  or  b  >  59  or  c  >  147  or  d  >  1 . 0 
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Non-specific  symptoms  of  Q  fever 

a.  Hx  of  Abdominal  pain 

b.  Hx  of  sore  eyes 

c.  Hx  of  malaise/fatigue 

d.  Hx  of  rash 

e.  Hx  of  myalgia/weakness 

f.  Hx  of  arthralgia 

g.  Hx  of  nausea  or /and  vomiting 

h.  Hx  anorexia 

i.  Hx  of  chest  pain 

j.  Hx  of  chills 

k.  Hx  of  cough 

l.  Hx  of  weight  loss  >  5% 

m.  PE  shows  clubbing 

n.  Hx  of  back  pain 

o.  Hx  of  headache 

p.  Hx  of  diarrhea 

q.  @ Anemia 

True  if  6  of  a-k  and  4  of  (1-q) 


References: 

1.  Dr.  Larry  Reimer 

2.  Raoult  et  al.  "Chronic  Q  ffever:  diagnosis  and  follow  up",  Ann.  New 
York  Academy  of  sciences,  51-60 

3.  TS  Marrie  "A  comparison  of  Qfever  endocarditis  with  native  valve 
endocarditis",  Ann.  New  York  Academy  of  sciences,  61-67 
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Disease  name:  Q  fever  (late) 

A  priori  =  .00001 

findings:  sensitivity  1-specificity 


a.  Hx  of  Q  fever 

.80  (1) 

.001 

or 

Hx  of  contact  with  cattle,  sheep,  goats 

.70 

.02 

or 

Hx  of  work  in  animal  research  facility 
housing  sheep 

.20 

.005 

b.  @Fever 

.90 

.15 

or 

@Non-specific  symptoms  of  Q  fever 

.50 

.20 

c.  Sex,  male 

.85 

.57 

e.  @Abnormal  liver  functions  criteria  for  Q  fever 

.25 

.05 

or 

PE  shows  hepatomegaly 

.25  (2) 

.06 

or 

PE  shows  splenomegaly 

.07  (2) 

.01 

f.  Complement  fixation  test  shows  phase  I 

.99 

.001 

antibody  titer  >  1:128  to  C.  Burnetii 
h.  @Culture-negative  endocarditis 

.95 

.0001 

References: 

1.  Dr.  Larry  Reimer 

2.  Clarke,  (see  above  references) 

3.  Dr. Thomas  Marrie 

a  priori:  about  1/3  y  around  here  or  about  1/100,000 
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Disease  name:  Culture-negative  endocarditis 
A  priori  =  .0001 

findings:  sensitivity  1-specificity 


a .  Heart  murmur 

.85 

.10 

or 

definite  change  in  murmur  or  new  murmur 

.10 

.001 

or 

@Aortic  insuffj cency 

.15 

.0011 

or 

QMitral  valve  prolapse 

.29 

.01 

or 

Hx  of  mitral  valve  prolapse 

.25 

.03 

b.  Blood  culture  positive  for  bacteria 

c.  vegetations  on  heart  valves 

.0000001 

.01 

( echocar di ography ) 

.50 

.02 

d.  Hx  of  a  heart  valve  replaced? 

.05 

.002 

or 

Hx  of  rheumatic  fever 

.20 

.01 

e.  @Infective  embolic  phenomena 

.50 

.001 

f.  joint  pains 

.33 

.10 

g.  @Anemia  of  chronic  disease 

.80 

.07 

References: 

1.  Dr.  Larry  Reimer 

2.  see  also  infective  endocarditis 
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